build: fix Bazel 0.20.0 compatibility issues #1661

wchargin · 2018-12-05T19:11:09Z

Summary:
Bazel 0.20.0 introduced some incompatible changes that broke both our
code and our dependencies. Our dependencies also introduced some
incompatible changes of their own. This commit upgrades our dependencies
and fixes up our code to accommodate them.

An upgrade to transitive dependency rules_go introduces a dependency
on the @bazel_tools//tools/build_defs/cc:action_names.bzl builtin
target. This target was only introduced in Bazel 0.16.0, so as a
consequence our workspace is no longer compatible with earlier versions
of Bazel. In practice, we must use at least version 0.16.1 to fix a
regression in 0.16.0: bazelbuild/bazel#5726.

Unfortunately, Bazel 0.16.0 also introduced a regression that prevents
us from using the --distinct_host_configuration=false option; this has
not been fixed as of Bazel 0.20.0. As a result, parts of our build will
now be twice as slow.

The new urllib3 dependency is due to a transitive Selenium upgrade.

This commit shifts our Bazel version window from [0.13.0, 0.19.2] to
[0.16.1, 0.20.0].

Test Plan:
Running bazel test //tensorboard/... passes in Python 2 and 3 in Bazel
versions 0.16.1 and 0.20.0 (latest at time of writing), with
the .bazelrc file set up as on Travis:

>~/.bazelrc
echo "startup --output_base=${HOME}/.bazel-output-base" >>~/.bazelrc
echo "startup --host_jvm_args=-Xms500m" >>~/.bazelrc
echo "startup --host_jvm_args=-Xmx500m" >>~/.bazelrc
echo "startup --host_jvm_args=-XX:-UseParallelGC" >>~/.bazelrc
echo "build --local_resources=400,2,1.0" >>~/.bazelrc
echo "build --worker_max_instances=2" >>~/.bazelrc
echo "build --worker_verbose" >>~/.bazelrc
echo "build --worker_sandboxing" >>~/.bazelrc
echo "build --spawn_strategy=sandboxed" >>~/.bazelrc
echo "build --genrule_strategy=sandboxed" >>~/.bazelrc
echo "test --test_verbose_timeout_warnings" >>~/.bazelrc
echo "build --verbose_failures" >>~/.bazelrc
echo "test --test_output=errors" >>~/.bazelrc

wchargin-branch: bazel-0.20.0-fixes

Summary: Bazel 0.20.0 introduced some incompatible changes that broke both our code and our dependencies. Our dependencies also introduced some incompatible changes of their own. This commit upgrades our dependencies and fixes up our code to accommodate them. One of the changes is the removal of `native.[new_]http_archive`, which is replaced by a rule to be loaded from the `@bazel_tools` namespace. This rule was only introduced in Bazel 0.16.0, so as a consequence our workspace is no longer compatible with earlier versions of Bazel. The new `urllib3` dependency is due to a transitive Selenium upgrade. This commit shifts our Bazel version window from [0.13.0, 0.19.2] to [0.16.0, 0.20.0]. Test Plan: Running `bazel test //tensorboard/...` passes in Python 2 and 3 in Bazel versions 0.16.0, 0.19.2, and 0.20.0 (latest at time of writing). wchargin-branch: bazel-0.20.0-fixes

wchargin · 2018-12-05T19:26:07Z

Travis failure is:

The command "wget -t 3 -O bazel https://mirror.bazel.build/github.com/bazelbuild/bazel/releases/download/${BAZEL}/bazel-${BAZEL}-linux-x86_64" failed and exited with 8 during .

I'll figure out how to mirror this.

wchargin · 2018-12-05T19:49:45Z

Uploaded Bazel to mirror; restarting build.

nfelt

Thanks so much for figuring this all out! How hard would it be to add the actual github release URL for bazel as fallback call to wget in travis.yml? That way we aren't solely dependent on the mirror.

.travis.yml

wchargin · 2018-12-06T07:48:17Z

The Travis failure, which I could not reproduce locally with the same
version of Bazel, appears to be due to a bug in Bazel 0.16.0 that
introduces an implicit dependency on the native Java runtime. Upgrading
to Bazel≥0.16.1 solves this issue.

It looks like there may be more issues to track down; I’ll investigate
further tomorrow.

wchargin-branch: bazel-0.20.0-fixes

WORKSPACE

wchargin-branch: bazel-0.20.0-fixes

wchargin · 2018-12-06T21:45:51Z

Thanks so much for figuring this all out! How hard would it be to add
the actual github release URL for bazel as fallback call to wget in
travis.yml? That way we aren't solely dependent on the mirror.

This sounds like a good idea. If you don’t mind, I’ll do it in a
follow-up PR.

(Edit: I didn’t answer your question. It’s not hard.)

wchargin-branch: bazel-0.20.0-fixes

wchargin · 2018-12-06T22:13:09Z

@nfelt: Does this PR still have your approval?

nfelt

LGTM

.travis.yml

wchargin-branch: bazel-0.20.0-fixes

@nfelt

Summary: As suggested by @nfelt in: <#1661 (review)> Test Plan: Verify that the new chunk of shell script has the following properties: - When executed with the environment variables at the top of the Travis config, successfully downloads Bazel 0.16.1 and returns 0. - When executed with `BAZEL=0.19.2` (which does not exist in the mirror) and shasum updated per the instructions in the comment, successfully downloads Bazel 0.19.2 and returns 0. - When executed with `BAZEL=0.wat.0` (which does not exist), fails and returns 1. - When a bit is flipped in the shasum, fails and returns 1, having removed the invalid downloads. Note that Shellcheck reports no errors or warnings with `#!/bin/sh`. wchargin-branch: travis-bazel-download-github-fallback

@nfelt

Summary: As suggested by @nfelt in: <#1661 (review)> Test Plan: Verify that the new chunk of shell script has the following properties: - When executed with the environment variables at the top of the Travis config, successfully downloads Bazel 0.16.1 and returns 0. - When executed with `BAZEL=0.19.2` (which does not exist in the mirror) and shasum updated per the instructions in the comment, successfully downloads Bazel 0.19.2 and returns 0. - When executed with `BAZEL=0.wat.0` (which does not exist), fails and returns 1. - When a bit is flipped in the shasum, fails and returns 1, having removed the invalid downloads. Note that Shellcheck reports no errors or warnings with `#!/bin/sh`. wchargin-branch: travis-bazel-download-github-fallback

This is mainly copy-paste from TensorBoard repo. Please see the PR for more details. tensorflow/tensorboard#1661

wchargin requested a review from nfelt December 5, 2018 19:11

wchargin added the type:build/install label Dec 5, 2018

nfelt approved these changes Dec 5, 2018

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

wchargin added 2 commits December 6, 2018 11:13

fixup: add versions.check

6673c9a

wchargin-branch: bazel-0.20.0-fixes

fixup: bump BAZEL=0.16.1

ced53d0

wchargin-branch: bazel-0.20.0-fixes

stephanwlee reviewed Dec 6, 2018

View reviewed changes

WORKSPACE Outdated Show resolved Hide resolved

wchargin added 4 commits December 6, 2018 12:31

stopship: Travis debugging

9389baf

wchargin-branch: bazel-0.20.0-fixes

stopship: more Travis debugging

79cb9ed

wchargin-branch: bazel-0.20.0-fixes

stopship: Bazel==0.16.1 with distinct host config

1d93c6a

wchargin-branch: bazel-0.20.0-fixes

fixup: unhacked version of working configs

ce0b2d7

wchargin-branch: bazel-0.20.0-fixes

fixup: bump versions.check to 0.16.1

c0acdd9

wchargin-branch: bazel-0.20.0-fixes

nfelt approved these changes Dec 6, 2018

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

.travis.yml Show resolved Hide resolved

wchargin added 2 commits December 6, 2018 14:27

review: add “keep in sync” comments

d73ff13

wchargin-branch: bazel-0.20.0-fixes

review: remove commented-out flag entirely

56c0f56

wchargin-branch: bazel-0.20.0-fixes

wchargin merged commit cdfcb45 into master Dec 6, 2018

wchargin deleted the wchargin-bazel-0.20.0-fixes branch December 6, 2018 22:39

wchargin mentioned this pull request Dec 6, 2018

travis: add GitHub fallback for Bazel download #1667

Merged

stephanwlee added a commit to stephanwlee/tensorboard-plugin-example that referenced this pull request Dec 17, 2018

Fix repo for Bazel 0.20.0

e171e83

This is mainly copy-paste from TensorBoard repo. Please see the PR for more details. tensorflow/tensorboard#1661

stephanwlee mentioned this pull request Dec 17, 2018

Fix repo for Bazel 0.20.0 tensorflow/tensorboard-plugin-example#45

Merged

stephanwlee added a commit to tensorflow/tensorboard-plugin-example that referenced this pull request Dec 18, 2018

Fix repo for Bazel 0.20.0 (#45)

8a7f215

This is mainly copy-paste from TensorBoard repo. Please see the PR for more details. tensorflow/tensorboard#1661

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: fix Bazel 0.20.0 compatibility issues #1661

build: fix Bazel 0.20.0 compatibility issues #1661

wchargin commented Dec 5, 2018 •

edited

Loading

wchargin commented Dec 5, 2018

wchargin commented Dec 5, 2018

nfelt left a comment

wchargin commented Dec 6, 2018

wchargin commented Dec 6, 2018 •

edited

Loading

wchargin commented Dec 6, 2018

nfelt left a comment

build: fix Bazel 0.20.0 compatibility issues #1661

build: fix Bazel 0.20.0 compatibility issues #1661

Conversation

wchargin commented Dec 5, 2018 • edited Loading

wchargin commented Dec 5, 2018

wchargin commented Dec 5, 2018

nfelt left a comment

Choose a reason for hiding this comment

wchargin commented Dec 6, 2018

wchargin commented Dec 6, 2018 • edited Loading

wchargin commented Dec 6, 2018

nfelt left a comment

Choose a reason for hiding this comment

wchargin commented Dec 5, 2018 •

edited

Loading

wchargin commented Dec 6, 2018 •

edited

Loading